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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Computer  Based  Instructional  Systems  Team  of  the  US  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  performs  research  and 
development  in  the  area  of  educational  technology  that  applies  to  military  train¬ 
ing.  Of  interest  are  methods  for  training  individuals  to  develop  and  utilize 
instructional  courseware  in  reasonable  time,  at  an  acceptable  cost, 

This  Technical  Report  describes  the  development  and  formative  evaluation 
of  eighteen  Job  Aids  designed  to  support  users  of  the  instructional  Systems 
Development  Model  (ISD,  TRADOC  Pamphlet  350-30).  The  £SD  Model  is  a  step-by-step 
procedure  for  the  analysis,  design,  development,  implementation,  and  control  of 
military  course  materials.  To  accomplish  this  research,  ARI's  resources  were 
augmented  by  contract  DAHC19-78-C-Q100  with  the  Hum^h  Resources  Research  Organi¬ 
zation. 

Personnel  at  the  US  Army  Engineer  School,  Fo^t  Belvoir,  Virginia,  provided 
guidance  during  the  developmental  phase  of  the  project;  Dr.  Everett  Rompf, 

Mr.  Jack  Ainsworth  and  Ms.  Lois  Passman.  In  addition,  the  following  individuals 
at  the  US  Army  Signal  Center  and  Fort  Gordon  gave  valuable  assistance  during  the 
needs  assessment  and  evaluation  phases:  Dr.  William  Ketner,  Mr.  Earl  Duncan, 

Mr.  Richard  Arnold,  Mr.  John  Meyers,  Ms.  Doris  Lakeman,  Mr,  Dan  Danilovich, 

Ms.  Yvonne  Waldin  and  Mr.  Richard  Eberl.  Ms.  Jean  Farrell  and  Mr.  Michael 
Hillelsohn  of  HumRRO  also  contributed  to  this  effort. 

The  entire  research  project  is  responsive  to  the  requirements  of  Army 
Project  2Q263743A794,  FY  80  Work  Program. 


PREFACE 


This  document  reports  on  a  research  project  in  which  18  Job  Aids  were  developed 
to  permit  implementation  of  all  phases  (analyze,  design,  develop,  implement,  and  control) 
of  the  Instructional  Systems  Development  (ISD)  model. 

The  research  effort  was  performed  in  HumRRO’s  Educational  and  Training  Systems 
Division,  Dr.  Robert  J.  Seidel,  Director.  Mr.  Russel  E.  Schulz  was  the  Project  Director. 

This  research  was  supported  by  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  under  Contract  No.  DAHC19-78-C-0010,  “Computer  Interactive 
Author  Aiding.”  The  project  was  monitored  technically  by  Dr.  Melissa  Berkowitz,  Dr.  Bruce 
Knerr,  Dr.  Harold  F.  O’Neil,  Jr.,  and  Dr.  Leon  Nawrocki  of  the  U.S.  Army  Research  Insti¬ 
tute.  Their  support  and  assistance  in  this  research  effort  is  gratefully  appreciated.  The 
authors  also  wish  to  express  gratitude  to  all  personnel  at  the  U.S.  Army  Engineer  School, 
Fort  Belvoir,  Virginia  and  the  U.S.  Army  Signal  Center  and  Fort  Gordon,  Fort  Gordon, 
Georgia  who  provided  assistance  in  the  research  project.  Special  appreciation  is  extended 
to  the  following  individuals  who  contributed  to  the  success  of  the  project: 

U.S.  Army  Engineer  School 

Dr.  Everett  Rompf,  Education  Specialist,  Individual  Training  Division 
Mr.  Jack  Ainsworth,  Education  Specialist,  Individual  Training  Division 
Ms.  Lois  Passman,  Education  Specialist,  Individual  Training  Division 

U.S.  Army  Signal  Center  and  Fort  Gordon 

Dr.  William  D.  Ketner,  Deputy  Assistant  Commandant  for  Educational  Technology 
Mr.  Earl  Duncan,  Education  Specialist,  Educational  Technology  Division 
Mr.  Richard  Arnold,  Education  Specialist,  Educational  Technology  Division 
Mr,  John  Meyers,  Former  Chief,  CE  Operations  Brunch,  Training  Analysis  Division 
Ms.  Doris  D.  Lakeman,  Chief,  CE  Operations  Branch,  Training  Analysis  Division 
Mr.  Dan  Danilovich,  Chief,  CE  Maintenance  Branch,  Design  and  Development  Division 
Ms.  Yvonne  S.  Walden,  Education  Specialist,  Design  and  Development  Division 
Mr.  Richard  Eberl,  Training  Specialist,  Design  and  Development  Division 

The  assistance  provided  by  Ms.  Jean  Farrell,  formerly  of  HumliRO,  who  co-authored 
many  of  the  Job  Aids  is  gratefully  acknowledged  as  is  the  work  of  Mr.  Michael  Hillelsohn 
who  authored  the  Job  Aitl  for  ISD  V.3  Revise  System.  Finally,  a  special  thanks  to 
Mrs.  Alice  Thompson  whoso  excellent  work  contributed  to  the  design  and  overall 
appearance  of  the  Job  Aids. 


DEVELOPMENT  OF  JOB  AIDS  FOR  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT 


BRIEF _ 

Requirement : 

The  purpose  of  the  research  was  to  develop  and  evaluate  Job  Aids  ("how  to 
do  it"  guidance)  for  the  activities  identified  in  the  Instructional  Systems 
Development  Model  (ISD,  TRADOC  Pamphlet  350-30).  The  usefulness  of  the  ISD  Model 
depends  upon  Job  Aids  which  will  enable  training  personnel  to  translate  the  ISD 
procedures  into  instructional  products. 

Procedure : 

A  series  of  Job  Aids  was  designed  and  developed  for  each  of  the  five  phases 
of  the  ISD  Model  based  on  needs  assessments  conducted  at  the  US  Army  Engineer 
School  (USAES) ,  Ft  Belvoir,  VA  and  the  US  Army  Signal  Center  and  Ft  Gordon 
(USASC&FG) ,  Ft  Gordon,  GA.  Baseline  data  were  collected  from  USAES  and  USASC&FG 
personnel  as  they  performed  their  ISD  functions  without  the  assistance  of  the 
Job  Aids.  The  Job  Aids  were  then  provided  to  individuals  who  were  engaged  in 
the  activity  covered  by  the  Job  Aid.  Data  were  collected  relating  to  problems 
alleviated  by  using  tie  Job  Aid,  problems  encountered  in  using  the  Job  Aid,  and 
user  acceptance  of  the  Job  Aid. 

Findings : 

Comparisons  were  made  with  the  baseline  data  to  assess  the  value  of  the 
Job  Aids  as  used  in  the  ISD  process.  The  findings  strongly  supported  the  use¬ 
fulness  of  the  Job  Aids. 

Utilisation  of  Findings: 

A  summative  evaluation  of  the  Job  Aids  is  planned  to  assess  their  value  to 
training  personnel  in  the  preparation  and  delivery  of  a  complete  course  of 
instruction. 
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INTRODUCTION 


The  purpose  of  this  section  is  to  provide  a  brief  description  of  research  and  develop¬ 
ment  in  this  area,  a  discussion  of  the  rationale  for  this  research,  and  an  overview  of  the 
project  history. 


A  SYSTEMS  APPROACH  TO  TRAINING 

The  systems  approach  defines  a  process  which  focuses  on  the  job  that  is  ultimately 
to  be  performed  and  upon  the  individual  who  is  to  learn  to  perform  that  job.  The  systems 
approach  is  just  what  the  name  implies:  a  systematic  process  for  specifying  the  desired 
products  of  training  and  selecting  what  will  be  taught,  how  it  will  be  taught,  what  the 
presentation  mechanism  will  be,  and  evaluating  the  effects  of  each  phase  of  the  process. 

It  focuses  on  student  performance  as  a  determinant  of  content.  Its  proper  application  can 
hardly  fail  to  improve  instruction  where  only  incidental  attention  has  been  given  to  these 
functions.  However,  after  more  than  two  decades  of  research  and  development  in  this 
area,  the  full  benefits  of  applying  the  systems  approach  to  instruction  have  yet  to  be 
realized. 

Karly  efforts  to  standardize  a  definitive  technology  in  this  area  included  those  by 
HumKRO  (Crawford.  19(12;  Pimm,  1962;  Vallunce  &  Crawford,  19621  on  behalf  of  the 
U. S.  Army  in  tin*  early  1960s  and  the  development  of  the  C.S.  Continental  Army  Com¬ 
mand  Regulation  350-100-1.  Systems  Engineering  of  Training,  (1968)  and  the  C.S.  Air 
Force  Manual  50-2.  Instructional  Systems  Development  (1970). 

These  attempts  Itave  culminated  in  the  Instructional  Systems  Development  (ISO) 

Model  (TRADOC  t*nm  350-30.  1975).  This  model  was  prepared  by  the  Center  for 
Educational  Technology  at  Florida  State  University  under  contract  with  the  Interservice 
Committee  for  Instructional  Systems  and  Development,  involving  the  Navy,  Air  Force, 
Marine  Corps,  and  the  Army.  The  ISD  contains  standardized  rationale,  terminology,  and 
basic  concepts  of  instructional  systems. 

Difficulties  arose  in  applying  the  ISO  approach  to  military  training.  Much  of  the 
early  research  in  attempting  to  overcome  these  problems  involved  attention  to  the  tech¬ 
nical  difficulties.  One  of  the  pvrimary  difficulties  of  the  1SI)  model  and  its  manuals  is 
that  it  emphasizes  “what  to  do.”  not  “how  to  do  it."  ISI)  manuals  art*  intended  to  have 
general  applicability.  However  the  same  methodologies  cannot  be  applied  to  the  universe 
of  training  problems.  "How  to  do  it”  may  vary  widely  even  though  the  specific  processes 
or  “what  to  do”  may  remain  fairly  constant  across  training  problems.  Initially,  the  “what 
to  do”  nature  of  the  ISD  manuals  was  not  a  severe  jirohlem  inasmuch  as  early  users  of 
the  manuals  wen*  “expert"  training  developers.  However,  later,  the  possibility  of  having 
laypersons  use  the  ISI)  approach  to  achieve  the  successes  of  the  experts  was  exploml. 

The  use  of  an  ISD  manual  by  military  personnel  with  little  or  no  experience  in  training 
program  design  may  cost  a  fraction  of  the  cost  of  hiring  or  contracting  out  to  experts  to 
do  Use  development.  However,  specific  guidance,  tools,  and  jirooeduros  are  needed  by 
such  laypersons  for  ISD  to  bo  appropriately  and  cost-effectively  implemented  witlitn  the 
military  training  establishment. 


THE  NEED  FOR  JOB  AIDS 


In  this  report  "how  to  do  it”  guidanee/tools/procedures  will  be  referred  to  as  Job 
Aids.  These  are  designed  to  help  all  members  of  an  instructional  staff  charged  with  meet¬ 
ing  the  objectives  of  one  or  more  components  of  the  instructional  systems  development 
process. 

Job  Aids  are  needed  in  all  Phases  of  the  ISD  process  if  significant  improvements  are 
to  occur  in  applying  this  systematic  approach  toward  producing  instruction.  This  approach 
is  expensive,  time  consuming,  critical,  and  requires  specialized  capabilities. 

A  wide  variety  of  aids  do  exist,  but  the  majority  are  more  of  the  “what  to  do"  type 
than  the  “how  to  do  it”  variety.  Rather  than  providing  actual  help  in  performing  the 
work,  or  even  providing  detailed  how-to-do-it  guidance,  most  of  the  existing  aids  may  serve 
to  reinforce  or  broaden  the  guidance  provided  in  the  ISD  manuals.  Aagard  and  Braby 
(19761  note  the  practical  limitations  in  using  their  algorithms  and  guidelines  for  general 
purposes,  “The  task  categories  and  related  guidelines  are  not  at  a  level  that  will  accommo¬ 
date  any  training  setting.”  Briggs'  handbook  (19701  may  also  serve  such  a  broad  guidance 
function. 

Care  should  be  exercised  in  the  selection  of  aids  to  be  integrated  with  ISD,  however, 
so  that  authors  are  not  confused  by  different  yet  similar  models,  sets  of  jargon,  proce¬ 
dures  of  forms.  Existing  general  manuals  differ  from  one  another  in  that  they : 

•  Include  different  steps  or  different  names  for  steps. 

•  Include  different  methods  of  accomplishing  each  step. 

•  1  Vo  vide  different  levels  of  specificity  in  the  details  included  under  each 
step. 

•  Provide  different  formats  for  reporting  the  work  accomplished  under 
each  step. 

Thus,  to  provide  the  most  efficient  aids  to  authors,  the  guidance  found  in  some  of  these 
manuals  and  guidebooks  needs  to  be  translated  and  integrated  into  the  ISD  framework, 
rather  than  referred  to  in  its  source  form. 

Recently,  increasing  emphasis  has  been  placed  on  the  development  of  “how  to  do 
it"  aids.  A  course  which  guides  trainees  in  specifying  learning  activities  and  in  developing 
tests  and  instructional  materials  is  reported  by  O’Neal  and  O'Neal  (1970).  Dyer,  Matthews, 
Wright  and  Vudowiteh  (1976)  prejami  guidance  for  instructional  technologists  who  are 
tasked  to  develop  or  administer  questionnaires  as  part  of  Army  field  tests  ami  evaluations. 
In  an  effort  to  assure  the  quality  of  instructional  development,  Kllis,  Wulfeck.  and 
Fredericks  (1979)  developed  the  Instructional  Quality  Inventory-  Conoley  and  O’Neil 
(1979)  presented  a  detailed  and  pragmatic  set  of  guidelines  for  multiple-choice  item 
generation  and  review. 

In  Uh>  project  described  in  this  document.  Job  Aids  were  developed  to  provide  the 
specific  guidance  ("how  to  do  it”)  needed  for  implementing  the  ISD  model. 
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PROJECT  HISTORY 

On-iine  Job  Aids  developed  in  a  previous  ARI  project  (Schulz,  Hibbits,  Wagner  & 
Seidel,  1979)  demonstrated  the  utility  of  assisting  authors  to  implement  the  ISD  process. 
The  major  characteristics  which  accounted  for  the  usefulness  of  the  Job  Aids  are: 

•  The  Aids  are  query-based  and  interactive. 

»  Ihey  are  based  upon  a  realistic  authoring  needs  assessment  at  an  operational 
site. 

•  The  Job  Aids  were  developed  and  evaluated  according  to  ISD  guidance  and 
recommended  procedures. 

•  Detailed  authoring  flowcharts  accompanied  on-line  Job  Aids. 

•  The  Job  Aids  were  designed  to  be  system  independent. 

•  The  Job  Aids  concentrate  on  assisting  users  to  deal  with  structural  and  adminis¬ 
trative  details  permitting  them  to  concentrate  their  efforts  on  the  subject 
matter  content  of  the  materials. 

The  research  project  described  in  this  report  is  an  extension  of  the  previous  ARI 
project.  In  the  present  research,  Job  Aids  were  developed  to  assist  Army  instructional 
systems  development  personnel  in  the  analysis,  design,  development,  implementation  and 
control  of  instructional  materials. 

The  research  began  in  November  1977.  During  the  research  period  Job  Aids  were 
developed  and  formatively  evaluated.  The  ISD  model  was  used  to  define  the  general 
activities  to  be  covered  in  the  Job  Aids.  (Figure  1  shows  the  ISD  model.)  Each  Job  Aid 
consists  of  two  volumes— Descriptive  Authoring  Flowcharts  and  a  Job  Aid  Manual.  The 
latter  contains  prescriptive  descriptions  of  the  activities  to  be  performed,  along  with 
critical  references.  They  were  designed  to  be  stand-alone,  step-by-step  procedural  guides, 
equally  useful  to  individuals  at  all  experience  levels  of  the  Instructional  Systems  Develop¬ 
ment  process.  The  Job  Aids  were  based  on  an  assessment  of  user  needs  at  the  U.S.  Army 
Engineer  School  (USAES),  Fort  Belvoir,  Virginia,  and  the  U.S.  Army  Signal  Center  and 
Fort  Gordon  (USASC&FG),  Fort  Gordon,  Georgia.  They  were  implemented  and  evaluated 
at  the  USASC&FG. 

Initially,  13  Job  Aids  were  developed  and  evaluated,  one  for  each  block  of  the  ISD 
model  except  for  Block  1.1,  Analyze  Job.  A  Job  Aid  was  not  developed  for  this  block 
for  the  following  reasons: 

(1)  The  research  plan  called  for  the  formative  evaluation  of  the  Job  Aids  at  the 
USAES.  Discussions  with  USAES  personnel  indicated  that  the  majority  of  jobs  at  the 
USAES  had  undergone  job  analysis.  Since  personnel  at  USAES  would  not  be  performing 
this  activity  during  the  research  period,  one  could  not  evaluate  a  Job  Aid  developed  for 
this  activity. 

(21  Job  analysis  is  a  highly  complex  activity  that  requires  input  from  many  sources. 
Many  of  these  sources  are  outside  of  the  USAES.  Therefore,  it  would  have  been  difficult 
to  obtain  the  input  in  a  form  needed  for  the  research  effort. 

(3)  Because  of  the  complexity  of  performing  job  analysis,  and  the  multiplicity  of 
Job  Aicu  needed  for  doing  so,  development  of  an  adequate  Job  Aid  for  this  block  would 
require  too  much  of  the  total  research  resources. 
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Figure  1.  THE  ISO  BLOCKS  IN  EACH  OF  THE  FIVE  ISO  PHASES 


The  evaluation  plan  for  tl~.  initial  13  Job  Aids  was  to  have  USAES  instructional 
development  personnel  use  the  Job  Aids  in  an  actual  job  setting  as  they  were  performing 
the  activities  required  by  the  ISD  model.  However,  shortly  after  beginning  work  it  was 
discovered  that  all  Phase  I  ISD  activities  for  courses  at  USAES  had  been  completed. 
Previous  information  indicated  that  all  ISD  Phase  I,  II,  and  III  activities  were  being  con¬ 
tinuously  performed  at  USAES.  Because  no  new  courses  were  to  be  established  during 
the  research  period  it  would  not  be  possible  to  conduct  a  formative  evaluation  of  the  Job 
Aids  for  Blocks  1.2  (Select  Tasks /Functions),  1.3  (Construct  Job  Performance  Measures), 
1.4  (Analyze  Existing  Courses),  and  1.5  (Select  Instructional  Setting)  at  USAES.  In  order 
to  accomplish  the  work,  a  revision  in  the  research  plan  was  made  to  change  the  location 
at  which  the  aids  were  to  be  evaluated.  The  U.'k  Army  Signal  Center  and  Fort  Gordon 
(USASC&FG),  Fort  Gordon,  Georgia  was  selected  as  a  substitute  for  USAES  for  the 
following  reasons: 

(1)  Activities  in  all  phases  of  the  ISD  model  were  being  actively  worked  on  at 
USASC&FG. 

(2)  The  avai1  ability  of  more  complete  data  at  USASC&FG  would  permit  more 
reliable  and  useful  Job  Aids  to  result  from  the  research  effort. 

(3)  Due  to  the  considerable  number  of  personnel  available  at  USASC&FG  who 
were  engaged  in  the  ISD  process,  it  would  be  feasible  to  collect  the  data  required  for 

the  formative  evaluation  in  a  timely  manner  requring  a  minimal  delay  in  project  activities. 

Upon  completion  of  the  development  and  formative  evaluation  of  the  Job  Aids  for 
the  first  three  phases  of  the  ISD  model  another  project  effort  was  initiated.  It  involved 
the  development  and  evaluation  of  a  prototype  system-independent.  Programming  Design 
Guide  for  an  interactive  Job  Aid.  The  research  was  performed  to  evaluate  the  feasibility 
of  translating  the  Job  Aids  into  a  computer-based  instruction  format. 

The  Programming  Design  Guide  provided  all  of  the  guidance  necessary  for  a  user  to 
implement  the  Job  Aid  on  any  of  a  large  number  of  computer  systems.  The  specific 
Job  Aid  selected  for  the  effort  was  the  Job  Aid  for  ISD  Block  1.5,  Select  Instructional 
Setting.  This  Job  Aid  was  selected  because  it  was  representative  of  all  of  the  Job  Aids 
in  the  first  three  phases  of  the  ISD  model  in  terms  of  level  of  difficulty  of  translation 
into  computer-based  instruction  format. 

The  Programming  Design  Guide  format  used  led  to  a  successful  implementation  of 
the  Job  Aid  for  selecting  instructional  settings.  As  this  Job  Aid  is  representative  of  the 
other  Job  Aids,  it  is  expected  that  the  approach  used  in  the  development  of  the  Pro¬ 
gramming  Design  Guide  will  be  equally  successful  for  implementing  other  Job  Aids 
on-line.  The  Programming  Design  Guide  research  is  described  in  a  report  by  Schulz, 
Underhill,  Hargan,  and  Wagner  (1980). 

The  successful  use  of  tho  13  Job  Aids  developed  earlier  in  this  project  for  ISD 
Phases  I,  II,  and  III,  indicated  that  the  interactive  job  aid  approach  would  also  meet 
the  requirements  for  Phases  IV  and  V  of  the  ISD  model.  Therefore,  five  additional 
Job  Aids  covering  the  activities  in  Phases  IV  and  V  were  developed.  Implementation 
and  evaluation  of  these  Job  Aids  occurred  at  the  USASC&FG. 

The  next  section  of  this  report  provides  a  detailed  description  of  the  analyses 
selection,  development,  and  formative  evaluation  of  the  18  Job  Aids  developed  during 
the  research  effort. 


RESEARCH  ACTIVITIES 


In  this  section  the  research  activities  involved  in  the  analysis,  selection,  development, 
and  formative  evaluation  of  the  18  Job  Aids  will  be  discussed.  Where  the  research  activ¬ 
ities  differed  between  the  development  of  the  first  13  Job  Aids  (Phases  I,  II,  and  III  of 
the  1SD  model)  and  the  last  5  Job  Aids  (Phases  IV  and  V)  the  research  activities  will  be 
described  separately.  (The  first  13  Job  Aids  will  hereafter  be  referred  to  as  Phase  I,  II 
and  III  Job  Aids.  Similarly  the  5  Job  Aids  for  Phases  IV  and  V  will  hereafter  be  referred 
to  as  Phase  IV  and  V  Job  Aids.)  The  development  of  the  Programming  Design  Guide  for 
on-line  implementation  of  the  ISD  1.5  Select  Instructional  Setting  Job  Aid  will  not  be 
discussed.  The  reader  is  referred  to  the  report  devoted  to  this  Guide  (Schulz,  Underhill, 
Hargan,  and  Wagner,  1980). 

The  research  effort  was  divided  into  three  major  tasks: 

•  Task  1:  Analysis  and  Selection  of  Job  Aids 

•  Task  2  Development  of  Job  Aids 

•  Task  3:  Evaluation  of  Job  Aids 

The  activities  involved  in  each  of  the  three  tasks  are  discussed  below. 

TASK  1:  ANALYSIS  AND  SELECTION  OF  IOB  AIDS 

Tas'-  1  activities  were  conducted  in  four  steps.  These  steps  are  (1)  perform  Job  Aids 
needs  assessment,  (2)  analyze  existing  job  aids,  (3)  identify  candidate  Job  Aids,  and  (4) 
refine,  modify  o:  complete  development  of  job  aids. 

Step  1.  Perform  Job  Aids  Needs  Assessment 

As  shown  in  Figure  1,  *he  ISD  Model  consists  of  19  major  ISD  activities  (blocks). 

In  the  Model,  each  of  these  activities  is  expanded  into  sub-element  activities.  For  example 
ISD  Block  II. 1  Develop  Objectives  has  t  .en  expanded  into  the  sub-element  activities 
shown  in  Figure  2. 

In  Step  1  a  detailed  Job  Aids  needs  assessment  was  performed  to  establish  priorities 
among  the  sub-elements  of  each  ISD  block  for  which  Job  Aide  were  to  be  completed. 
These  priorities  were  needed  because  time  restraints  imposed  upon  the  research  would 
not  permit  equal  developmental  empaasis  on  all  sub-element  activities. 

The  needs  assessment  for  the  Phu  1  I,  II  and  III  Job  .uds  was  performed  by  means 
of  a  detailed  structured  interview  administered  to  11  course  developers  and  10  training 
analysts  at  the  USAES.  A  copy  of  ttr  Deeds  Assessment  Inventory  used  for  the  inter¬ 
views  is  contained  in  Appendix  A.  The  interview  dealt  with  topics  such  as  the  following: 

»  ISD  products  produced  by  the  individual. 

♦  Tbe  ISD  input  received  by  the  individ”*!  and  problems  associated  with 

use  of  this  input. 
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Figure  2.  FLOWCHART  OF  BLOCK  11.1:  DEVELOP  OBJECTIVES 
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•  The  ISD  output  produced  by  the  individual  and  problems  associated  with 
its  production. 

•  ISD  training  available  to  the  individual  and  additional  training  required. 

•  Availability  of  existing  job  aids  and  when  available,  the  usefulness  of  the 
job  aid. 

•  Ideas  for  needed  additional  Job  Aids. 

The  results  of  the  interviews  indicated  that  there  was  a  considerable  need  for  Job 
Aids  in  all  sub-elements  of  the  blocks  within  the  first  three  ISD  phases.  Specific  needs 
were  identified  for  each  of  the  ISD  blocks  except  for  ISD  Block  II.3  (Describe  Entry 
Bahavior)  for  which  interviewees  could  not  identify  any  specific  sub-elements  in  need  of 
assistance. 

The  major  needs  for  Job  Aids  identified  in  the  structured  interviews  were  incorpo¬ 
rated  into  a  Needs  Assessment  Review  Form  which  was  then  administered  to  75  instruc¬ 
tional  development  personnel  at  the  USAES  and  33  development  personnel  at  the 
USASC&FG.  For  each  need  listed  on  the  Review  Form  the  respondents  were  asked  to 
rate  whether  they  “Strongly  Agree,”  “Agree,”  “Disagree,”  “Strongly  Disagree,”  or, 
“Cannot  Judge.”  A  copy  of  the  Needs  Assessment  Review  Form  is  contained  in 
Appendix  B. 

Major  findings  from  the  Needs  Assessment  Review  Form  which  shaped  the  content 
of  the  Job  Aids  are  as  follows: 

(1)  Specific  guidance  is  needed  for  completing  all  ISD  activities. 

(2)  Examples  of  the  products  are  needed. 

(3)  Sources  of  information  necessary  for  implementing  the  ISD  Model  (e.g., 
where  to  obtain  “frequency  of  task  performance”  data)  are  needed. 

The  Needs  Assessment  conducted  for  the  Phase  IV  mid  V  Job  Aids  was  conducted 
only  at  the  USASC&FG  with  27  instructional  development  personnel  who  were  currently 
working  in  the  ISD  Blocks  for  Phase  IV  and  V.  The  specific  questions  covered  for  each 
ISD  Block  are  shown  in  Appendix  C.  The  major  results  of  this  needs  assessment  are 
as  follows: 

—  ISD  IV.l  Implement  Instructional  Management  Plan.  Aids  are  needed  which 
provide  mathematical  formulas  for  determining  resources  needed  for  imple¬ 
mentation  of  the  management  plan.  Guidance  is  needed  for  reviewing  and 
revising  the  Instructional  Management  Plan. 

—  ISD  IV. 2  Conduct  Instruction.  The  instruction  that  instructors  receive  cur¬ 
rently  for  this  ISD  block  is  adequate  and  as  a  consequence,  there  is  no  need 
for  a  Job  Aid  for  the  actual  conduct  of  instruction.  It  was  agreed  that  a 
Job  Aid  for  administration  and  analysis  of  student  opinions  concerning 
instruction  would  be  helpful. 

—  ISD  V.l  Conduct  Internal  Evaluation.  The  USASC&FG  currently  uses  a 
checklist  for  conducting  internal  evaluation.  Instructional  guidance  for 
answering  the  questions  contained  on  the  checklist  was  considered  to  be 
needed. 

—  ISD  V.2  Conduct  External  Evaluation.  Respondents  expressed  the  need  for 
general  guidance  in  performing  an  external  evaluation. 

—  ISD  V.3  Revise  System.  No  respondent  was  able  to  identify  a  dear  need  for 
a  specific  Job  Aid  for  this  ISD  Block. 


Step  2.  Analyze  Existing  Job  Aids 

In  Step  2  ISD  literature  was  searched  to  identify  existing  job  aids  that  would  be 
useful  for  the  research  effort.  Two  major  sources  of  job  aid  information  was  TRADOC 
Pamphlet  350-30,  Interservice  Procedures  for  Instructional  Systems  Development  (1975) 
and  the  McDonnell  Douglas  Corporation  Bibliography  of  Aids  for  Course  Material  Devel¬ 
opment  (Logan,  1976).  Each  job  aid  identified  was  carefully  analyzed  to  determine: 

•  Its  relevancy. 

•  How  it  fits  into  the  sub-element  blocks. 

•  How  it  feeds  information  into  other  ISD  blocks. 

•  The  degree  of  similarity  with  other  job  aids. 

•  How  it  might  be  combined  with  other  job  aids  to  form  a  more  compre¬ 
hensive  Job  Aid. 

•  The  amount  of  the  total  research  effort  that  would  be  required  to  complete 
the  job  aid  if  it  were  selected  for  inclusion  in  the  final  listing  of  the  Job 
Aids  to  be  completed. 

In  general,  no  existing  job  aid  was  identified  that  could  easily  be  translated  into  the 
interactive  format  proposed  for  the  Job  Aids  to  be  developed  in  the  research  effort.  How¬ 
ever,  the  literature  search  did  produce  information  that  was  cataloged  for  later  use  in  Job 
Aid  development.  For  example,  much  of  the  ISD  materials  produced  at  the  USASC&FG 
later  proved  to  be  extremely  valuable  in  the  Job  Aid  development  effort. 

Step  3.  Identify  Candidate  Job  Aids 

Based  on  the  Job  Aid  needs  assessment  conducted  in  Step  1  and  the  analysis  of 
existing  aids  performed  in  Step  2,  candidate  job  aids  were  identified  in  Step  3. 

Step  4.  Refine,  Modify,  or  Complete  Development  of  Aids 

It  had  been  anticipated  that  existing  aids  would  be  identified  that  would  be  useful  in 
the  Job  Aid  research  effort.  It  was  thought  that  those  aids  would  in  most  cases  require 
only  minor  modification  for  presentation  in  an  interactive  format.  Tins  modification 
was  to  be  accomplished  in  this  Step.  However,  as  already  noted  in  the  discussion  of 
Step  2,  no  job  aids  were  identified  that  could  be  easily  modified  for  presentation  in  an 
interactive  format.  As  a  consequence,  tins  step  of  tire  research  was  not  performed. 

Rather,  the  information  contained  within  some  existing  job  aids  was  incorporated  into 
the  Job  Aids  developed  in  this  project. 


TASK  2:  DEVELOPMENT  OP  JOB  AIDS 

Previous  research  (Schulz,  1975a,  1975b,  Schulz,  Hibbits,  Wagner  &  Seidel  1979) 
indicated  the  utility  of  preparing  Job  Aids  that  make  use  of  a  flowchart  format.  While  the 
Job  Aids  developed  in  this  project  were  intended  to  be  used  initially  off-line  in  a  printed 
version,  they  were  to  be  developed  so  as  to  facilitate  possible  later  translation  into  on-line 
computer  versions.  Therefore,  each  Job  Aid  developed  in  this  project  is  composed  of  two 
documents—Descriptive  Authoring  Flowcharts  and  Job  Aid  manuals.  In  addition,  there  is 
a  document  which  provides  an  introduction  to  the  use  of  the  Job  Aids  and  general  infor¬ 
mation  regarding  useful  resources  for  ISD  implementation. 
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Job  Aid  Format 


Introduction  to  the  Use  of  Job  Aids  and  Job  Aids  Resource  Manual.  The  introduc¬ 
tion  describes  the  Job  Aids,  how  they  were  developed,  and  provides  general  information 
on  how  to  use  the  Job  Aids.  The  Job  Aids  Resource  Manual  provides  general  information 
concerning  CODAP  (Comprehensive  Occupational  Data  Analysis  Programs),  conducting 
field  surveys,  and  the  use  of  various  panels  of  personnel  in  the  ISD  process.  Figure  3 
shows  the  major  topics  included  in  the  Introduction  to  the  Use  of  Job  Aids  and  Job  Aids 
Resource  Manual. 

Descriptive  Authoring  Flowcharts.  The  Descriptive  Authoring  Flowcharts  are  the 
primary  documents  used  in  the  Job  Aids.  They  direct  the  user  to  specific  guidance, 
examples  and  references  provided  in  the  Job  Aid  Manuals.  Figure  4  shows  a  representa¬ 
tive  page  of  descriptive  authoring  flowcharts.  The  flowcharts  are  easy  to  use  in  that  they 
include  only  five  flowchart  symbols.  The  symbols  are  standard  flowcharts  symbols  except 
for  the  one  that  looks  like  an  open  book  or  manual.  This  symbol  is  used  to  cue  the  user 
that  he/she  is  to  go  to  the  Job  Aid  Manual  for  additional  information  or  guidance.  The 
directions  contained  within  the  flowchart  symbols  are  frequently  supplemented  by  narra¬ 
tive  information  provided  in  the  margin  of  the  Descriptive  Authoring  Flowcharts.  This 
narrative  information  is  sufficient  to  provide  an  overview  of  the  activities  that  are  to  be 
performed  in  the  Job  Aid.  The  Descriptive  Authoring  Flowcharts  are  also  sufficient  to 
act  as  an  executive  summary  for  supervisors  or  commanders. 

Job  Aid  Manuals.  As  stated  above,  the  Job  Aid  Manuals  provide  the  specific  guid¬ 
ance,  examples,  and  references  necessary  to  help  the  user  produce  the  products  for  the 
ISD  activities  covered  by  the  Job  Aid.  In  addition,  each  Job  Aid  Manual  has  associated 
with  it  one  or  more  worksheets  to  use  in  the  development  of  each  product.  When  the 
user  is  referred  to  the  Job  Aid  Manual  an  attempt  has  been  made  to  provide  four  types 
of  information:  (1)  the  purpose  for  performing  the  activity  called  for  in  the  flowchart 
block,  (2)  sources  of  information  for  performing  tire  activity,  (3)  specific  step-by-step 
guidance  for  performing  tire  activity,  and  (4)  if  the  activity  involves  filling  in  a  worksheet, 
an  example  of  what  the  worksheet  looks  like  after  the  activity  has  been  performed.  In 
addition,  tho  flowchart  block  for  which  guidance  is  given  is  reproduced  at  the  top  of  the 
page  of  the  Job  Aid  Manual.  Figure  5  shows  a  representative  page  from  a  Job  Aid  Manual. 
(Note  that  flowchart  block  9  is  also  represented  in  Figure  4.) 

As  previously  stated,  the  Job  Aids  were  intended  to  alleviate  the  needs  expressed 
during  the  needs  assessment  survey,  lir  some  cases,  guidance  is  readily  available  from 
other  sources  for  meeting  the  needs;  or  in  a  few  corses,  time  constraints  placed  on  the 
research  project  would  not  permit  Ure  development  of  some  needed  guidance.  As  a  con¬ 
sequence,  the  Job  Aids  do  not  always  provide  complete  guidance  or  information  for  all 
activities  in  tho  ISD  Block  covered  by  Ure  Job  Aid.  Therefore,  a  brief  summary  of  Ure 
Job  Aids  for  each  ISD  Block  is  provided  below. 

The  Job  Aids  for  the  first  tlrree  Plrases  of  tho  ISD  Model  are  available  as  Army 
Research  Institute  documents  (Schulz  &  Farrell,  1980a,  1980b,  1980c,  1980d,  1980e, 
1980f).  Job  Aids  for  the  remaining  phases  are  in  jrress  at  this  time  (Schulz,  Wagner  & 
Hillelsohn).  The  Job  Aids  are  grouped  by  ISD  Phase.  That  is,  there  is  a  separate  Descrip¬ 
tive  Authoring  Flowchart  manual  for  the  analysis,  design,  development,  implementation, 
and  control  phase  of  the  ISD  model.  Similarly,  there  is  a  Job  A'd  Manual  for  each  of  tire 
phases. 
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INTRODUCTION  TO  THE  USE  OF  JOB  AIDS 

What  are  Job  Aids? 

What  is  the  Classification  System  Used  in  TRADOC  Pamphlet  350-30  and 
Job  Aids? 

What  Sources  of  Information  Were  Used  in  the  Development  of 
Job  Aids? 

What  Job  Aids  are  Presently  Available  and  What  Do  They  Consist  of? 

What  are  the  Main  Parts  of  the  Flowchart  Manual? 

What  Flowchart  Symbols  are  Used  in  the  Flowchart  Manual? 

What  Instruction  is  Provided  Within  the  Flowchart  Symbols? 

What  is  the  Purpose  of  the  Supplemental  Information  Provided  in  the 
Flowchart  Manual? 

What  is  the  Flowchart  Block  and  Page  Numbering  System? 

How  Do  I  Use  the  Job  Aid  Manual  and  What  are  the  Main  Parts  of  it? 

What  is  the  Purpose  of  the  Partial  Flowcharts  at  the  Top  of  Some  Pages 
of  the  Job  Aid  Manual? 

What  is  the  Purpose  of  the  Questions  Written  in  Script  That  Appear  on 
Some  Pages  of  the  Job  Aid  Manual? 

What  is  the  Purpose  of  the  Completed  or  Partially  Completed  Examples 
of  Worksheets? 

What  Does  This  All  Mean  and  What  Do  I  Do  Now? 

JOB  AIDS  RESOURCE  MANUAL 

How  Can  the  Resource  Manual  Help  Me? 

What  Are  Some  Sources  of  Job  Significant  Data? 

Which  Sources  of  Data  are  Included  in  the  Resource  Manual? 


Figure  3.  MAJOR  TOPICS  INCLUDED  IN  THE  INTRODUCTION  TO  THE 
USE  OF  JOB  AIDS  AND  JOB  AIDS  RESOURCE  MANUAL 


II 


•■"'Vs*-* 


•  Task  inventory  includes  all  tasks  performea 
>n  this  skill  level. 


Guidance  lor  Obtaining 
Task  Inventory  list 


J 


•  Task  inventory  shou'd  be  available  as  output 
trom  ISO  l.l  Analyze  Job. 

•  if  procedures  lor  listing  tasks  by  Skill  level 
were  not  complete,  see  persons  responsible 
lor  compiling  task  list. 

•  When  list  has  been  obtained  return  to  Manual. 
ISO  1.2  Select  Tasks  lor  Training.  Slock  6, 
page  A-M. 


e  Critical  task  identification  and  final  selections  are  recorato  on  the  Cntical 
Task  Identification  Sheet  !CTI).  The  CTI  Sheet,  therefore,  provides  a 
documented  record  for  these  decisions  and  the  bans  for  seen. 

•  for  guidance  m  obtaining  CTl  Sheets,  sea  Manual,  page  A  s  2. 


Obtain  Information 
Necessary  lor  Selling 
Up  CTl  Sheet 


•  tnlotmei<on  needed  'o'  "tatting  up''  the  CTl  Sheet  includes  the  tallowing 

-  MOSeode 

-  Skill  level 

-  Task  IQ  numbers 

-  Task  inventory 

*  for  sources  dl  mlo/mauon  see  Manual,  page  A-IJ. 


>0 _ S 

j  Record  an  CTl  Sheet 
i  Date.  Pip  Number, 

r  Name,  MOS  Cede. 

Skill  Level.  Teak 
Numbers. 

Attxh  Tau  invemory 

- - - , - 

i 
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•  Number  each  CTl  Sheet  iDie  as  many  is  nectary  to  Nil  ail  the  tasksi 

•  Attach  a  copy  ot  the  task  inventory  to  the  hist  CTt  Sheet  to  neio  vov 
identify  the  tasks  to  which  the  numbers  rttir 

•  Example  See  Manual.  P*$*  A-ta. 


Figure  4.  REPRESENTATIVE  PAGE  FROM  DESCRIPTIVE 
AUTHORING  FLOWCHARTS 


from  block  8 


OfcWA  IfttKmttM 
Nttwrj  fw  S«ttw* 
UgCH  S2wt 


to  block  10 


•  Intormttwft  mum  tor  "nme*  a#'*  the  CT1  Shoot  mi dudea  On  (oUowuh: 
•  UQScoOt 

-  SkiHtMt 

-  T«fc  IQ  numbers 

-  Takiiwontory 

•  f  or  aaurtat  of  mtortMOOa  m  Mow. 


7&i ieU  cU  *)  teed  <utd  client  cU  *)  U? 

•  The  following  information  is  needed  before  you  can  set  up  the  CTI  Sheet.  Sources 
for  locating  each  item  of  information  are  identified  and  in  most  cases  throughout 
this  manual  the  source  will  be  rated  as  “excellent,”  “good,"  or  “fair.”  Under¬ 
standably  not  all  sources  of  information  will  be  readily  available  to  you. 


Information  Rewired 

Sourcell) 

Remark! 

MOS  and  Skill  Level 
designation 

•  Supervisor 

•  AR  611-1  (for  a  new  MOS1 

•  Output  of  ISO  1.1  Analyze  Job 

•  Excellent  source 

•  Excellent  source 

•  Excellent  source 

Talk  I.O.  Number! 

•  Numbering  system  prescribed  in 
TRAQOC  Circular  351-28,  if  already 
assigned 

•  Excellent  source 

•  Temporary  numbering  system  of  your 
own  design,  e.g.  1  through  500,  if 
authorited  numbers  have  not  been 
assigned 

•  Excellent  source 

Talk  inventory 

•  Output  of  ISO  1.1  or  job  analysis 

•  List  currently  in  use 

•  Excellent  source 

•  Good  source 

Figure  6.  REPRESENTATIVE  PAGE  FROM  JOB  AID  MANUAL 


PHASE  I  -  ANALYZE 


ISD  1.2  Select  Tasks/Functions 

The  Job  Aid  for  this  block  assists  the  user  in  making  the  decision  of  which  tasks  are 
to  be  selected  for  training  and  which  are  to  be  rejected.  The  output  from  using  the  Job 
Aid  is: 


•  A  final  list  of  tasks  for  training. 

•  A  list  of  tasks  rejected. 

»  A  summary  of  the  data  upon  which  the  selection  decision  was  based. 

ISD  1.3  Construct  Job  Performance  Measures 

This  Job  Aid  provides  guidance  for  performing  a  task  analysis  of  each  of  the  tasks 
selected  for  training.  The  output  from  use  of  the  Job  Aid  is: 

•  Information  concerning  the  conditions  under  which  each  task  is  performed. 

•  Cues  indicating  when  it  is  performed. 

•  Standards  indicating  how  well  it  should  be  performed. 

•  References  and  training  tips. 

•  Identification  of  task  elements  or  performance  steps. 

This  information  is  recorded  hr  a  format  which  can  be  placed  in  the  Soldier's  Manual, 
understood  by  the  soldier  leanring  the  task,  and  used  as  an  evaluation  tool  by  someone 
supervising  the  task. 

ISO  1.4  Analyze  Existing  Courses 

The  purjrose  of  this  Aid  is  to  eliminate  duplication  of  effort  in  several  ISD  processes. 
Hr  us.  it  contributes  to  one  of  the  overall  goals  of  ISD— to  increase  the  cooperative  devel¬ 
opment  and  use  of  training  throughout  tire  military  services.  Specific  guidance  is  provided 
for  analyzing  existing  courses.  The  output  from  this  Job  Aid  can  be: 

•  The  generation  of  an  initial  task  list  for  a  particular  MOS. 

•  The  identification  of  suitable  courses  for  trailring. 

•  The  identification  of  courses,  or  parts  of  courses,  which  may  be  suitable 
after  revision  by  course  development  personnel. 

ISD  1.6  Select  Instructional  Setting 

The  purpose  of  tire  Job  Aid  for  this  ISD  Block  is  to  provide  guidance  in  choosing 
instructional  settings  (training  locations)  fot  tasks  selected  for  training.  Use  of  the  Job 
Aid  will  result  in  a  listing  of  all  critical  tasks  in  which  each  task  is  assigned  for  training 
to  one  of  the  following  instructional  settings: 

•  Institution  (resident  school  training). 

•  Supervised  on-Ure*jiob  training  (SOJT). 

•  Self-study. 
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PHASE  It  -  DESIGN 


ISD  11.1  Develop  Objectives 

The  purpose  of  the  Job  Aid  for  this  ISD  Block  is  twofold.  First,  it  guides  the 
user  in  the  actual  writing  of  correctly  stated  learning  objectives.  Second,  it  explains  the 
mechanics  of  deriving  learning  objectives  from  Task  Summary  Sheets  (developed  in  ISD 
1.3  Construct  Job  Performance  Measures)  by  means  of  learning  analysis.  The  outputs 
from  use  of  this  Aid  are: 

•  Terminal  learning  objectives  for  each  task  to  be  trained. 

•  Learning  objectives  for  task  elements. 

ISD  11.2  Develop  Tests 

In  this  Job  Aid  the  user  is  provided  guidance  in  deciding  whether  tasks  and  task 
elements  should  be  tested  by  means  of  written  or  performance  type  test  items.  Infor 
mation  and  guidance  is  also  provided  for  the  preparation  of  written  test  items.  The  Job 
Aid  does  not  cover  the  development  of  performance  tests,  nor  does  it  cover  preparation 
of  administrative  materials  or  validation  of  test  items.  The  output  consists  of  written 
test  items  which  will  be  used  to  test  the  ability  of  soldiers  to  correctly  perform  critical 
tasks  or  task  elements. 

ISD  11.3  Describe  Entry  Behavior 

The  goal  of  the  Job  Aid  for  this  ISD  Block  is  to  provide  the  guidance  necessary  to 
test  the  entry  behavior  of  trainees  and  to  utilize  this  information  to  adjust  the  beginning 
point  of  instructional  units;  and  to  verify  or  revise  the  assumptions  made  in  the  learning 
analysis  performed  in  ISD  II. 1.  Develop  Objectives,  about  entry  skills  and  knowledges  to 
be  trained.  The  output  from  using  this  Job  Aid  is  a  validated  learning  analysis  for  each 
task. 

ISD  11.4  Determine  Sequence  and  Structure 

This  Job  Aid  provides  guidance  for  sequencing  and  grouping  learning  objectives  to 
help  assure  that  when  instructional  materials  are  developed,  each  teaming  objective  is 
placed  in  the  best  relationship  with  other  leanring  objectives  to. 

•  Produce  the  most  learning  in  the  shortest  period  of  time. 

•  Help  the  soldier  make  the  transition  from  one  skill  or  body  of  knowledge 
to  another. 

•  Assure  that  supporting  (enabling)  knowledges  and  skills  arc  acquired  before 
subsequent  subject  matter  is  introduced. 

Use  of  the  Job  Aid  will  result  in  the  proper  sequencing  of  all  tasks  and  task  elements. 


PHASE  III  -  DEVELOP 


ISD  1 11.1  Specify  Learning  Events/Activities 

The  purpose  of  the  Job  Aid  for  this  ISD  Block  is  to  help  the  user  to  select  appro¬ 
priate  learning  events  and  activities  for  training  sessions  in  which  critical  tasks  will  be 
taught.  Learning  research  has  identified  learning  activities  and  conditions  which  are 
essential  for  teaching  different  kinds  of  learning  objectives.  In  this  Job  Aid,  the  user 
learns  what  those  activities  and  conditions  are.  Use  of  this  Job  Aid  will  result  in  a  listing 
of  learning  activities  that  could  be  used  for  teaching  the  learning  objectives  for  each  criti¬ 
cal  task. 

ISO  1 1 1.2  Specify  Instructional  Management  Plan  and  Delivery  System 

The  Job  Aid  for  this  ISD  Block  enables  the  user  to  apply  a  systematic  approach  to 
the  selection  of  delivery  systems  (media)  for  presenting  instruction  to  trainees.  This 
systematic  approach  to  media  selection  requires  consideration  of  the  nature  of  the  objec¬ 
tives:  their  learning  categories/subcategories,  learning  activities,  and  instructional  settings. 
Such  factors  as  equipment  and  facility  constraints,  cost  of  procurement,  cost  of  replace¬ 
ment,  and  development  costs  are  also  considered  in  the  Job  Aid.  However,  the  Job  Aid 
does  not  provide  guidance  for  conducting  a  cost-effectiveness  analysis  of  media,  nor  for 
the  actual  development  of  the  Instructional  Management  Plan.  The  output  from  this  Job 
Aid  is  a  complete  identification  of  the  characteristics  of  a  delivery  system  to  be  employed 
for  teaching  each  task  and  task  element. 

ISD  1 1 1.3  Review/Select  Existing  Materials 

In  the  Job  Aid  developed  for  tiiis  ISD  Block  guidance  is  provided  to  help  instruc¬ 
tional  developers  evaluate  existing  materials  in  order  to  determine  their  usefulness  in  the 
training  program  Urey  are  preparing,  lire  Job  Aid  results  in  a  list  of  all  existing  materials 
ulrich  have  been  considered  for  use  in  the  course  development  activity.  Each  item  of 
materia)  is  evaluated  as  to  wheUrer  it  should  be  accepted  or  rejected  for  use  in  the  course. 
If  the  material  is  to  be  accepted,  suggestions  for  revisions  (if  needed)  are  documented. 
Each  item  of  material  is  Urerefore  classified  as: 

•  Accept  for  use  wrUrout  revision, 

•  Revise  for  use. 

•  Reject. 

ISD  111.4  Develop  Instruction 

lire  Job  Aid  for  this  ISD  Block  has  two  goals.  Hu*  main  goal  of  the  Job  Aid  is  to 
f-dst  in  producing  training  materials  for  the  following  media:  audio-only,  audio-visual, 
and  written  text.  A  secondary  goal  is  to  familiarize  Ure  user  with  the  ways  itr  which  Uu* 
various  production  personnel  can  help  in  the  instructional  development  effort.  The 
product  resulting  from  use  of  the  Job  Aid  is  a  first  draft  of  instructional  materials. 


ISD  IH.5  Validate  Instruction 


Guidance  for  conducting  two  types  of.  instructional  validation  is  provided  in  the  Job 
Aid  for  this  block  of  the  iSD  mr  ■•el.  They  are:  individual  validation  trials  and  group 
validation  trials.  Validation  as  used  in  this  block  of  the  ISD  is  the  process  by  which 
instructional  material  (course,  module,  lesson,  etc.)  is  tested  and  revised  until  it  teaches 
what  it  was  intended  to  teach.  The  Job  Aid  defines  each  of  these  types  of  trials  and 
provides  guidance  in  when  and  how  to  use  them.  The  outcome  from  use  of  this  Job 
Aid  is  a  set  of  validated  instructional  units. 


PHASE  IV  -  IMPLEMENT 

ISD  IV.1  Implement  Instructional  Management  Plar, 

The  purpose  of  the  Job  Aid  developed  for  this  ISD  Block  is  to  provide  guidance  in 
the  techniques  to  use  for  reviewing  and  revising  (when  necessary )  the  Instructional  Manage¬ 
ment  Plan  developed  in  ISD  III. 2.  The  output  from  the  use  of  this  Job  Aid  is  an  accurate 
Instructional  Management  Plan  suitable  for  implementation  in  ISD  IV. 2  Conduct 
Instruction. 

ISD  IV.2  -  C  "duct  Instruction 

Tb  s  Job  Aid  emphasizes  the  collection  of  student  performance  and  survey  data 
during  me  conduct  of  instruction.  It  does  not  cover  the  actual  conduct  of  instruction, 
inasmuch  us  this  guidance  is  available  from  a  number  of  other  sources.  Student  Survey 
Forms  and  Instructor  Survey  Forms  are  provided  which  permit  students/instructors  to 
provide  input  as  to  needed  changes  in  the  instructional  program.  The  Job  Aid  provides 
guidance  as  to  when  th*  survey  forms  should  be  administered  and  how  to  summarize 
data  received  from  them.  The  output  from  this  Job  Aid  will  be  student/instructor  data 
useful  for  determining  needed  changes  in  the  instructional  program.  Student/instructor 
data  are  addressed  in  the  internal  evaluation  report  prepared  in  ISD  Block  V.l. 


PHASE  V  -  CONTROL 

ISD  V.l  Conduct  internal  Evaluation 

The  purpose  of  the  Job  Aid  for  this  ISD  Block  is  to  provide  the  user  with  guidance 
in  conducting  an  internal  evaluation  of  courses  which  were  developed  using  the  Job  Aids 
described  above.  The  Job  Aid  assists  the  user  to  answer  the  following  questions: 

•  “Is  the  instruction  providing  the  students  with  the  necessary  knowledge 
and  skills  to  meet  the  objectives  in  a  satisfactory  manner?" 

•  “To  what  extent  did  the  development  effort  conform  to  the  ISD  process?" 

•  “How  good  are  the  products  resulting  from  the  various  ISD  activities?" 

A  detailed  checklist  and  guidance  for  its  use  is  provided  for  gathering  the  information 
needed  to  answer  the  above  questions.  The  output  from  this  Job  Aid  is  an  Internal 
Evaluation  Report  which  is  used  to  modify  (where  necessary)  the  products  from  previous 
ISD  Blocks. 
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ISD  V.2  Conduct  Externa!  Evaluation 

The  purpose  of  the  Job  Aid  for  this  ISO  Block  is  to  help  the  user  to  conduct 
external  evaluations  of  trainin'.;  programs.  Emphasis  is  given  in  the  Job  Aid  to  determine 
whether  a  schooi-taught  course  adequately  trains  its  graduates.  Graduate  and  Supervisor 
Survey  Forms  are  provided  in.  the  Job  Aid  and  instructions  are  given  as  to  how  these  are 
to  be  administered  and  how  the  data  derived  from  the  survey  are  to  be  summarized.  The 
output  from  use  of  this  Job  Aid  will  be  an  External  Evaluation  Report  which  will  be  used 
in  the  next  ISD  Block,  V.3  Revise  System,  to  determine  revisions  needed  in  the  instruc¬ 
tional  program. 

ISD  V.3  Revise  System 

Guidance  is  provided  in  this  Job  Aid  for  making  a  careful  analysis  of  data  relating 
to  doctrine  changes,  soldiers’  performance  on  Skill  Qualification  Tests  (SQT),  and  instruc¬ 
tional  program  deficiencies  reported  in  the  Internal  and  External  Evaluation  Reports. 
Algorithms  are  provided  for  determining  required  course  revisions  based  on  the  above 
information.  The  outcome  from  this  Job  Aid  will  be  a  report  detailing  the  revisions 
needed  in  the  instructional  program. 

The  :ob  Aids  described  above  were  formatively  evaluated  at  the  USASC&FG.  The 
formative  evaluation  was  conducted  in  Task  3,  Evaluation  of  Job  Aids.  A  discussion  of 
Task  3  follows. 


TASK  3.  EVALUATION  OF  JOB  AIDS 

Evaluation  of  Job  Aids  -  Laval  1  {Informal  Review  of  Known  Aids) 

Tire  first  level  evaluation  in  this  project  was  carried  out  as  part  of  the  Job  Aid  devel¬ 
opment  effort,  and  is  described  more  fully  in  Task  1.  Project  staff  reviewed  published 
guidance  and  known  ISD  aids,  attended  workshops  on  the  use  of  aids  such  as  the  Instruc¬ 
tional  Quality  Inventory  (IQ1)  (Ellis,  et  al„  1979),  and  visited  a  number  of  TRADOC 
installations  to  learn  about  any  useful  forms,  instruments,  or  guidance  that  should  be 
incorporated  into  the  Job  Aids. 

Evaluation  of  Job  Aids  -  Laval  2  (Formative  Evaluation  of  New  Aids) 

Previous  information  indicated  that  all  ISD  Phase  l-IH  activities  were  being  continu¬ 
ously  performed  at  the  USAES  at  Ft.  Bel  voir,  Va.  However,  in  early  visits  to  USAES,  it 
was  discovered  that  it  would  not  be  possible  to  eotleci  evaluation  data  for  ISD  Phase  J 
Jot  .Aids  inasmuch  as  there  was  no  one  who  would  be  working  in  this  area  during  die 
proposed  evaluation  period.  The  USASC&FG  at  Ft.  Gordon,  Ga.,  was  chosen  as  Uie 
alternative  location,  and  evaluation  plans  were  modified  to  accommodate  the  new  site. 

In  Task  1,  those  ISD  activities  in  most  need  of  procedural  guidance  and  assistance 
were  identified.  It  was  against  these  needs  that  the  Job  Aids  were  evaluated.  The  focus 
of  this  evaluation  was  whether  or  not  the  problems  noted  during  the  needs  assessment 
were  solved  or  ameliorated  by  the  Job  Aids. 

A  detailed  formative  evaluation  plan  was  prepared  which  set  up  procedures  for 
gathering  evaluation  data.  The  evaluation  plan  was  described  to  USASC&FG  personnel, 
and  approval  obtained  prior  to  its  implementation. 
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One  of  the  features  of  the  evaluation  plan  was  the  collection  of  baseline  data.  Its 
purpose  was  to  determine  how  well  school  personnel  were  performing  ISD  Phase  I-III 
activities  without  the  assistance  of  the  Job  Aids  to  be  provided  in  this  project.  A  copy 
of  the  instrument  used  for  the  collection  of  baseline  data  is  shown  in  Appendix  D. 

Following  the  collection  of  baseline  data  the  Job  Aids  were  provided  to  Ft.  Gordon 
personnel  to  use  in  their  ISD  activities.  USASC&FG  personnel  were  given  a  very  short 
training  session  in  the  purpose  and  use, of  the  Job  Aids.  It  was  emphasized  in  the  train¬ 
ing  session  that  the  focus  of  the  evaluation  was  the  Job  Aid  itself,  not  the  person  who 
was  working  with  it.  Job  Aid  deficiencies  were  to  be  pointed  out,  but  so  too  were  those 
aspects  of  the  Job  Aid  that  were  deemed  to  be  helpful  (e.g.,  more  efficient  procedures 
or  specific  guidance,  etc.). 

After  each  individual  completed  using  the  Job  Aids  he/she  was  interviewed  concern¬ 
ing  the  practicality  and  usefulness  of  the  Job  Aids.  The  structured  interview  form  used 
for  these  interviews  is  shown  in  Appendix-  E.  In  addition  to  interviewing  Job  Aid  users, 
their  immediate  supervisors  were  also  interviewed.  In  this  latter  interview  the  major 
emphasis  was  the  supervisor’s  perceptions  of  any  reduction  or  impiovement  in  the 
efficiency  of  their  subordinate’s  work  from  the  use  of  the  Job  Aids.  Anally,  all  person¬ 
nel  (Job  Aid  users  and  their  immediate  supervisors)  were  asked  to  identify  specific 
changes,  deletions,  or  additions  that  would  increase  the  accuracy,  timeliness,  and  useful¬ 
ness  of  the  Job  Aids.  Any  serious  problems  that  were  identified  were  attended  to  as 
soon  as  possible  and  revisions  made  to  the  Job  Aids. 

In  the  evaluation  of  Phase  IV  and  V  Job  Aids,  “expert”  review  was  the  main 
approach  used  in  assessing  the  Job  Aids  developed  for  these  ISD  Phases.  Opinions 
regarding  user  acceptance  were  also  obtained,  as  was  information  on  problems  that  these 
personnel  expected  to  experience  while  using  the  Job  Aids. 

Evaluation  of  Job  Aids  —  Level  3  (Impact  on  Instruction) 

The  Job  Aids  were  designed  to  facilitate  many  steps  in  the  ISD  process—and  as  such, 
ultimately  should  have  an  effect  on  instructional  quality  and  relevancy.  Those  criteria 
are  measured  by  a  change  in  the  behavior  of  students  who  undergo  training  in  the  course 
resulting  from  the  ISD  process.  Although  accumulation  of  student  data  could  have 
indirectly  served  to  assess  the  adequacy  of  the  Job  Aids,  it  was  decided  that  this  activity 
not  be  performed  as  part  of  the  formative  evaluation  of  the  Job  Aids.  Rather,  more 
attention  was  paid  to  tire  determination  of  Job  Aids'  usefulness  and  the  need  for 
improvements. 

Findings 

The  formative  evaluation  plan  proposed  in  tins  project  produced  much  useful 
information  for  the  revision  and  improvement  of  Die  Phase  MU  Job  Aids.  The  plan 
was  followed  for  the  Phase  MH  Job  Aids  to  the  extent  that: 

—  Baseline  data  were  obtained  from  USASC&FG  personnel  regarding  their 
ISD  activities  before  Job  Aids  were  introduced. 

—  Acceptance  and  use  data  were  obtained  following  intervention  with  the  Job 
Aids  from  some  of  the  personnel  from  whom  baseline  data  were  collected. 
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The  data  obtained  by  the  procedures  outlined  above  were  examined  to  determine 
the  acceptance  and  utility  of  the  Job  Aids,  and  to  see  if  any  improvements  needed  to  be 
made  that  would  increase  their  efficiency  and/or  effectiveness.  The  effects  of  the  Job  Aids 
were  most  readily  observed  when  the  same  USASC&FG  personnel  who  participated  in 
Task  1  (needs  assessment)  were  the  ones  from  whom  the  data  were  obtained  regarding 
their  use  of  the  Job  Aids.  Data  obtained  from  others  highlighted  the  generality  of  the 
Job  Aids’  usefulness. 

Background  data  were  collected  from  personnel  at  U.S.  Army  Signal  Center  and  Fort 
Gordon,  Fort  Gordon,  Ga.,  who  were  engaged  in  activities  covered  by  ISD  Phases  I-J.II,  Back¬ 
ground  Information  Forms  and  ISD  Activity  Checklists  (see  Appendix  D)  were  admin¬ 
istered  to  over  200  individuals.  Approximately  60  of  these  individuals  were  visited  one 
week  later  and  interviewed  fay  project  staff  concerning  their  ISD  activities  during  the 
preceding  two-week  period.  The  specific  individuals  interviewed  were  selected  on  the 
basis  of  the  ISD  Activity  Checklists  completed  the  week  before.  In  this  way,  a  compre¬ 
hensive  set  of  information  was  gathered  regarding  the  job  activities  of  personnel  working 
in  the  ISD  process.  The  problems  they  encountered,  the  time  to  perform  certain 
activities,  and  their  opinions  regarding  the  effectiveness  and  efficiency  of  the  ISD 
process  (prior  to  the  intervention  of  Job  Aids),  were  among  the  topics  of  most  interest 
that  were  accumulated  in  the  baseline  data.  Similar  sets  of  data  were  obtained  following 
their  use  of  the  Job  Aids. 

The  results  of  this  evaluation  were  unexpected.  First,  the  intensity  and  extent  of 
needs  expressed  by  the  USASC&FG  and  US  AES  personnel  was  much  greater  than  had 
been  anticipated.  As  was  described  in  Task  I,  Job  Aids  were  perceived  as  needed  in  all 
blocks  of  ISD  Phases  Mil,  and  covered  practically  all  activities  within  each  block.  These 
needs  were  supported  by  the  baseline  data  collected  in  the  ISD  activity  sheets.  These, 
data  showed  that  the  ISD  activities  took  considerable  time,  due  in  part  to  the  inaccurate 
and  untimely  information  generated  by  inexperienced  individuals.  With  this  situation 
as  the  baseline  condition,  the  reaction  to  the  Job  Aids  was  judged  as  positive. 

All  (100%)  of  the  individuals  given  the  Job  Aids  thought  they  were  Good  or 
Excellent.  The  only  negative  comments  were  editorial  in  nature.  Problems  with  the 
flowchart  format,  as  seen  by  its  developers,  were  not  confirmed  by  the  users.  Follow-up 
structured  interviews  (see  Appendix  E)  with  57  of  the  users,  elicited  so  much  support 
for  the  draft  Job  Aids,  that  a  change  in  project  plans  seemed  essential  For  example, 
there  was  the  refusal  by  some  of  the  users  to  part  with  the  draft  Job  Aids  on  which  their 
editorial  comments  were  written,  unless  they  could  be  replaced  with  other  copies.  Also, 
copies  of  the  Job  Aids  were  requested  by  other  personnel  in  the  school  who  were  not 
participants  in  this  study,  but  wunted  to  use  them  in  their  own  ISD  work.  In  addition, 
the  Phase  I  and  11  Job  Aids  are  being  incorporated  into  a  USASC&FG  course  that 
teaches  the  analysis  and  design  of  instructional  development.  This  is  only  some  of  the 
evidence  of  acceptance  that  influenced  the  change  in  direction  taken  in  the  project. 

Evaluation  plans  were  curtailed  and  resources  shifted  to  Job  Aid  development  which 
permitted  a  broader  set  of  activities  to  be  covered  by  each  Job  Aid  than  had  been 
proposed  initially. 

In  the  formative  evaluation  of  Phase  IV  and  V  Job  Aids  the  data  emphasized 
editorial  changes  to  the  documents.  While  it  is  true  that  some  of  the  same  personnel  who 
responded  in  Urn  needs  assessment,  evaluated  the  Phase  Mil  Aids,  this  was  not  the  case 
for  the  Phase  IV  and  V  Job  Aids.  Due  to  time  and  resource  constraints,  the  ultimate  users 
were  not  able  to  participate  in  the  evaluation.  Rather,  only  “expert”  critiques  of  tire 
draft  Aids  were  obtained  and  the  Job  Aids  modified  accordingly.  However,  these  Phase 
IV  and  V  Job  Aids  will  be  sent  to  USASC&FG  for  review  prior  to  their  release  by  AIU 
for  general  use. 


RECOMMENDATIONS 


A  major  shortcoming  of  the  evaluation  was  the  fact  that  the  continuous  nature  of 
the  ISD  process  was  not  accommodated.  That  is,  each  Job  Aid  was  prepared  for  use  as 
a  “stand-alone”  set  of  documents  that  would  assist  those  engaged  in  the  activities  covered 
by  a  given  ISD  block.  It  was  assumed  that  the  information  feeding  into  that  block— 
the  information  necessary  for  using  a  particular  Job  Aid— would  be  correct  and  available 
in  a  timely  manner.  This  meant  that  the  output  or  products  of  one  Job  Aid  would  have 
to  be  available  for  use  by  those  who  needed  that  product  as  input  to  their  own  Job 
Aid.  Thus,  when  the  evaluation  of  Phases  I-III  Job  Aids  was  planned  to  occur  simul¬ 
taneously,  an  adequate  evaluation  of  the  impact  or  utility  of  the  Job  Aid  was  made 
extremely  difficult  and  impractical  within  the  constraints  of  this  project’s  time  and  resources. 
As  one  result,  the  focus  of  the  evaluation  shifted  from  a  comparison  of  pre-  and  post-use 
measures  of  the  Job  Aids,  to  a  detailed  critique  and  debugging  of  the  Job  Aids  by  actual 
and  potential  users.  There  is  still  the  need  to  evaluate  the  utility  and  impact  of  the  Job 
Aids  when  applied  to  the  ISD  process  as  a  whole. 

First,  the  Job  Aids  need  to  be  consolidated.  There  are  many  examples  of  redundancy 
and  duplication  of  forms  and  effort  that  should  be  eliminated. 

Second,  the  Job  Aids  should  be  used  throughout  the  entire  ISD  process  for  a  given 
course  development  effort.  In  this  way,  a  realistic,  longitudinal  evaluation  of  the  Job 
Aids  can  be  accomplished.  The  contribution  of  each  Job  Aid  to  the  ISD  process,  and  the 
eventual  impact  upon  instructional  quality  and  relevancy,  could  be  assessed  in  this  type 
of  evaluation. 
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APPENDIX  A 


NEEDS  ASSESSMENT  INVENTORY  FOR  ISD  PHASES  I,  II,  AND  III 
JOB  AID  DEVELOPMENT 


I.  IDENTIFICATION 

1.  Name: _ _ _ 

2.  Title: _ _ _ 

3.  Organization: _ _ _ 

4.  Are  you  a  subject-matter  expert  in  any  military  area(s)?  If  so,  what  area(s)? 


5.  Have  you  ever  been  an  instructor  at  USAES?  If  so,  what  courses,  positions, 
when,  how  long,  etc.? 


6.  How  long  have  you  worked  in  ISD? 


II.  JOB 

1.  In  which  USAES  courses  do  you  work  or  have  responsibility  for  someone 
olse's  work? 


2.  What  are  your  responsibilities? 


3.  In  what  ISD  activities  are  you  presently  working  or  supervising  others? 
How  long  for  each? 


4.  What  additional  ISD  areas  have  you  worked  on  (or  supervised  in)  during  the 
past  year?  How  long  for  each? 


What  did  you  produce? 


III.  INTERFACE 


1.  From  whom  do  you  (or  the  individuals  you  supervise)  receive  input? 


a.  What  problems,  if  any,  do  you  (or  your  staff)  have  in  using  the  input? 


b.  How  much  time  per  day /week/month  do  you  (or  your  staff)  spend  on 
the  related  tasks  because  of  the  problem? 


c.  What  control  do  you  have  over  the  quality  of  the  input? 


d.  Do  you  have  any  suggestions  as  to  how  this  input  oould  be  improved? 
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What  are  the  Output  products  resulting  from  your  (or  the  individuals  you 
supervise)  ISD  activities? 


a.  To  whom  (or  ISD  block/activity)  are  the  products  delivered? 


b.  What  feedback  do  you  receive  concerning  the  quality  of  the  products? 


c.  (Supervisors  only)  How  frequently  do  you  send  work  back  to  your 
staff  for  revision?  Explain. 


d.  (Non-supervisors)  How  frequently  does  your  supervisor  send  your  work 
back  for  revision?  Explain. 


IV.  TRAINING  AND  JOB  AIDS 


1.  What  training  have  you  had  (military  or  civilian)  on  subjects  related  to 
instructional  systems  development,  educational  technology,  and  the  like? 


2.  What  additional  training,  if  any,  would  you  like  to  receive? 


3.  What  job  aids  are  available  to  you  or  your  staff  for  performing  ISD  activities? 


a.  Which  of  these  do  you  actually  use? 


b.  How  useful  are  the  job  aids? 


c.  How  could  they  be  improved? 


a 


4.  What  locally  produced  forms  do  you  (or  your  staff)  complete  relevant  to 
ISD  activities? 


5.  Which,  if  any,  of  these  forms  do  you/your  staff  use?  (Show  interviewee 
appropriate  forms  shown  in  ISD  manual.) 


6.  What  happens  to  the  completed  form?  (How  is  it  used,  by  whom,  etc.?) 


V.  OTHER  PROBLEMS 


1  What  other  problems  do  you  encounter  in  your  work  that  lave  not  been 
mentioned  earlier? 


2. 


Do  you  have  ideas  about  possible  improvements  in  the  ISD  process  that  would 
help  solve  this  problem(s)? 


VI.  IDEAS  FOR  ADDITIONAL  AIDS 

1.  If  you  had  the  choice  of  selecting  five  ISD  aids  for  development,  what  ISD 
areas  would  they  cover? 


2.  Do  you  know  someone  who  can  provide  additional  ideas  on  the  subjects  we 
have  discussed? 
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APPENDIX  B 


NEEDS  ASSESSMENT  REVIEW  FORM  FOR  PHASES  I,  II,  AND  III 
JOB  AID  DEVELOPMENT 


NAME: _ 

TITLE: _ Telephone  Number: 

ORGANIZATION  (Division,  branch,  etc.!: _  _ 


How  long  have  you  worked  in  ISC? 


********** 

Job  Aids  are  being  prepared  for  individuals  who  are  performing  the  activities  called 
for  in  TRADOC  Pamphlet  350-30  (Interservice  Procedures  for  Instructional  Systems  Devel¬ 
opment),  Phases  I.  II,  and  III.  These  aids  will  be  given  to  USAES  personnel  to  try  out  and 
evaluate  in  the  months  ahead. 

In  January  and  February  1978  some  USAES  individuals  from  ITaDD  and  ODD  were 
surveyed  as  to  the  kinds  of  help  they  need  in  their  work.  The  statements  presented  on 
the  following  pages  represent  the  outcome  of  this  survey 

We  would  like  :r  you  to  read  each  statement  and  check  whether  you  Agree  or  Dis¬ 
agree  with  it.  The  scale  presented  has  four  points  so  that  you  can  indicate  the  level  of 
your  agreement  or  disagreement.  If  you  cannot  judge  a  particular  statement  because  you 
have  not  worked  in  that  phase  of  1SD,  check  the  box  labeled  “Cannot  Judge.”  Please 
check  the  *  Cannot  Judge”  box  only  when  absolutely  necessary.  A  page  has  also  been 
provided  for  you  to  add  other  assistance  you  need  in  your  work. 

Thank  you  for  your  cooperation.  Your  response  to  the  statements,  and  the  added 
comments,  will  provide  greater  assurance  that  the  aids  that  we  develop  will  indeed  provide 
the  needed  assistance  to  you  in  your  job. 


GENERAL 

Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

Cannot 

Judge 

1.  We  need  complete  approved  examples 
of  the  products  we  are  to  produce. 

□ 

□ 

□ 

□ 

□ 

2.  We  need  help  in  finding  sources  of 
information. 

□ 

□ 

/ 

□ 

□ 

□ 

3.  We  need  a  general  orientation  to  the 
course  development  process. 

□ 

□ 

a 

□ 

□ 

LSD  BLOCK  1.2:  SELECT  TASKS/FUNCTIONS 

4.  We  need  help  in  identifying  and 
defining  standards  for  tasks. 

□ 

□ 

□ 

□ 

□ 

5.  More  information  is  needad  on  con¬ 
ditions  of  task  performance. 

□ 

□ 

□ 

□ 

□ 

G.  We  need  assistance  in  selecting  critical 
tasks  at  the  appropriate  skill  levels. 

□ 

□ 

□ 

□ 

□ 

1.  Examples  of  good  task  statements  are 
needed. 

□ 

□ 

□ 

□ 

□ 

8.  Clearly  spoiled  out  procedures  ate  needed 
for  identifying  and  describing  critical 
tasks  and  associated  performance 
standards. 

□ 

D 

□ 

□ 

□ 

ISO  BLOCK  1.3-  CONSTRUCT  JOB 
PERFORMANCE  MEASURES 

9.  More  accurate  information  is  needed 
tegarding  the  sequence  of  steps  in  task 
performance. 

□ 

□ 

□ 

□ 

□ 

10.  Realistic  information  is  needad  regarding 
actual  job  performance,  available  equip¬ 
ment,  end  references. 

□ 

□ 

□ 

a 

□ 

11.  More  complete  end  dettiled  Information  is 
needed  regarding  task  performance  require¬ 
ments  to  include  enabling  skiiis/knowladgtv 

□ 

□ 

□ 

0 

□ 
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ISD  BLOCK  1.4:  ANALYZE 
EXISTING  COURSES 


Strongly 

Agree 


Agree 


Strongiy 
Disagree  Disagree 


Cannot 

Judge 


12. 

Documentation  is  needed  on  previous 
training  analyses  and  decisions. 

□ 

□ 

□ 

□ 

□ 

ISD  BLOCK  1.5:  SELECT 

INSTRUCTIONAL  SETTING 

13. 

More  guidance  is  needed  on  how  to  select 
from  all  possible  training  environments. 

□ 

□ 

□ 

□ 

□ 

14. 

We  need  more  guidance  on  the  purpose, 
content  and  format  of  Soldier's  Manuals. 

□ 

□ 

□ 

□ 

□ 

ISD  BLOCK  11.1:  DEVELOP  OBJECTIVES 

15. 

More  guidance  is  needed  on  how  to  prepare 
learning  objectives. 

□ 

□ 

□ 

□ 

□ 

16. 

We  need  guidance  on  how  to  identify  and 
analyte  a  task's  skill  and  knowledge 
components. 

□ 

□ 

□ 

□ 

□ 

17. 

Examples  of  appropriately  identified  skills 
and  knowledges  are  needed. 

□ 

□ 

□ 

□ 

n 

ISD  BLOCK  11.2:  DEVELOP  TESTS 

18. 

Wa  need  a  standard  format  for  preparing 
sor's. 

□ 

□ 

□ 

□ 

□ 

19. 

We  need  a  redesigned  "Brown  Book" 
containing  examples  of  SQTs. 

a 

□ 

□ 

□ 

□ 

20. 

There  is  a  need  for  aids  in  preparing 
course  criterion  tests. 

□ 

□ 

□ 

□ 

□ 

21. 

We  need  guidance  on  how  to  develop  seif- 
evaluation  checklists  and  scoresheau. 

□ 

□ 

□ 

□ 

□ 

22. 

Guidance  on  now  to  write  good  test  ques¬ 
tions  is  needed. 

□ 

0 

□ 

□ 

□ 

23. 

Guidance  it  needed  on  how  to  datormina 
the  difficulty  of  test  questions. 

□ 

□ 

□ 

□ 

□ 
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ISO  BLOCK  11.3:  DESCRIBE  strongly 

ENTRY  BEHAVIOR  A9ree 

Agree 

Disagree 

Strongly 

Disagree 

No  Aids  ware  recommended 

ISD  BLOCK  11.4:  DETERMINE 
SEQUENCE  AND  STRUCTURE 

24.  Examples  of  course  maps  are  needed. 

□ 

□ 

n 

□ 

25.  Examples  of  prescriptive  outlines  are  needed. 

□ 

□ 

□ 

.□ 

ISD  BLOCK  111.1:  SPECIFY  LEARNING 
EVENTS/ACTIVITIES 

26.  We  need  hslp  io  identify  the  critical 
instructional  materials. 

□ 

□ 

□ 

□ 

iSD  BLOCK  III. 2:  SPECIFY  INSTRUCTIONAL 
MANAGEMENT  PLAN  AND  DELIVERY 
SYSTEM 

27.  We  need  guidance  on  how  to  select  media 
appropriate  to  specific  objectives. 

□ 

□ 

□ 

□ 

28.  Guidance  is  needed  on  how  to  develop 
an  instructional  management  plan. 

□ 

□ 

□ 

□ 

ISD  BLOCK  111.3:  REVIEW/SELECT 
EXISTING  MATERIALS 

29.  We  need  guidance  on  how  to  locate,  select, 
end  ravitw  existing  materials. 

□ 

□ 

□ 

□ 

ISD  BLOCK  111,4:  DEVELOP  INSTRUCTION 

30.  Guidance  Is  needed  on  the  selection,  us>,  and 
evaluation  of  visuals. 

□ 

□ 

□ 

□ 

31.  Aids  ere  needed  for  storyboarding  sound- 
tUde  lessons. 

□ 

□ 

□ 

□ 

32.  Guidance  it  needed  on  how  to  write  effectively. 

D 

□ 

□ 

c 

33.  Aids  ere  needed  for  obtaining  end  using 
student  date  to  identify  weaknesses  m 
training  materials. 

□ 

□ 

a 

□ 

ISO  BLOCK  lll.B:  VALIDATE  INSTRUCTION 

fio  Aids  wart  recommended. 


Cannot 

Judge 


□ 

□ 

□ 

□ 

□ 

Q 

Q 

Q 

O 

□ 
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APPENDIX  C 


NEEDS  ASSESSMENT  INVENTORY  FOR  ISD  PHASES  IV  AND  V 
JOB  AID  DEVELOPMENT 

1V.1  IMPLEMENT  INSTRUCTIONAL  MANAGEMENT  PLAN 
Sample:  Chief  (or  his  Representative)  of  Each  Academic  Department 


QUESTIONS 


1.  Name 

2.  Academic  department 

3.  Courses  taught  in  department— student  load  and  frequency  of  classes 

4.  For  which  courses  did  DTD  provide: 

a.  front-end  analysis  (Phase  1) 

b.  design  products  (Phase  II) 

c.  course  development  products  (Phase  III) 

d.  instructional  management  plan 

e.  instructor’s  manual 

f.  student’s  manual 
*  POI 

h.  instructional  content 

5.  Were  modifications  needed  in  any  of  the  products  in  4  above?  If  yes,  who  made 
these  changes?  (That  is,  did  you  send  them  back,  or  do  it  yourself?) 

6.  If  not  provided  by  DTD,  are  the  products  in  4  above  available?  Who  prepared 
them? 

7.  If  DTD  did  not  provide  airy  of  the  products  listed  in  4  above: 

Why  didn't  they?  Organizational  problems?  Time,  etc.? 

8.  What  job  aids  are  available  for  the  implementation  of  the  instructional  management 
plan? 

9.  What  additional  job  aids  would  be  useful? 


IV.2  CONDUCT  INSTRUCTION 


Sample:  From  each  academic  department,  as  appropria>',  one  or  two  instructors 
selected  on  the  basis  of: 

1.  Type  of  instruction  presented,  e.g.,  self -packed,  CBI,  conventional,  etc. 

2.  Instructional  experience 

3.  Position  within  instructional  team 

4.  Student  load 


QUESTIONS 

1.  Name 

2.  Course  taught 

3.  Instructional  responsibility 

4.  Student  load 

a.  per  class 

b.  per  year 

c.  How  much  time  is  there  between  classes  for  lesson  changes? 

5.  Instructional  experience-type,  length  of  time,  etc. 

6.  Subject-matter  experience 

7.  Who  developed  your  instruction? 

8.  How  frequently  are  changes  made  in: 

a.  POl 

b.  Instructor’s  Manual 

c.  Student’s  Manual 

d.  instruction:-  content 

e.  records  maintained 

f.  r-tudent  load 

g.  lesson  plans 

9.  Who  makes  the  ciianges  in  tire  materials  above? 
i .  Are  changes  fed  beck  to  DTD? 

10.  How  frequently  does  management  plan  change  due  to: 

a.  unexpected  increase/decrease  in  student  loan 

b.  equipment  shortages 

c.  environment 

d.  supplies  and  materials 

e.  availability  of  instructors 


u.  rn,t  forma  are  you  required  to  MH.  whom?-eoPy  available. 

12.  What  job  aids  are  available  to  you?  (FM  s) 

13.  What  aids  would  be  useful  to  you  in  presenting  instruction? 


V.1  CONDUCT  INTERNAL  EVALUATION 


Sample:  Representative  From  Internal  Evaluation  Group. 


QUESTIONS 


1.  Name 

2.  Department 

3.  Is  there  an  intern'd  evaluation  conducted  for  each  course  for  which  the  signal  school 
has  propanency?  If  so: 

a.  Who  conducts  it? 

b.  When  is  it  conducted— how  frequently? 

c.  What  are  the  main  steps  of  the  evaluation? 

d.  What  guidance  is  available  (TRADOC,  local  SOP,  other  (get  copies). 

e.  What  standard  forms  are  used?  (Get  copies) 

f.  What  other  data  is  collected? 

g.  How  is  the  data  summarized? 

h.  How  are  the  results  used?  Who  are  they  sent  to? 

L  How  do  the  results  feed  into  other  ISD  Phases? 

(1)  If  not  used  what  are  the  reasons? 

(a)  timeliness 

(b)  they  have  their  own  source 

(c)  format 

(d)  other 

4.  What  problems  exist,  especially  between  divisions? 

5.  How  does  ISD  V.l  differ  from  IIL5? 

6.  What  aids  would  be  useful? 
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V.2  CONDUCT  EXTERNAL  EVALUATION 
V.3  REVISE  SYSTEM 


Sample:  One  or  more  individuals  from  Directorate  of  Evaluation 


QUESTIONS 


1.  Name 

2.  Who  determines  the  evaluation  procedures  used?  TRADOC,  local  SOP,  other. 

3.  If  mailed  questionnaires  are  used  in  the  evaluation: 

a.  What  is  the  sample  size? 

b.  Who  is  sampled?  e.g.,  graduate,  supervisor,  etc. 

c.  How  soon  after  graduation  is  individual  surveyed? 

d.  Are  questionnaires  MOS  specific?  If  yes; 

(1)  Are  copies  available? 

(2)  What  is  the  level  of  task  description  contained  on  questionnaire? 

If  no,  obtain  copy  of  general  questionnaire. 

e.  How  is  the  questionnaire  administered  (e.g.,  mailed  to  individual,  included 
in  201  file,  mailed  to  unit,  etc.)? 

f.  What  control  procedures  are  used  to  insure  questionnaire  return? 

g.  What  is  the  rate  of  return? 

h.  Are  there  any  follow-up  procedures? 

i.  Prior  to  graduation,  what  information  is  provided  student  concerning  field 
feedback  (i.e.,  answering  questionnaire)? 

4.  For  on-site  evaluation  visits 

a.  What  MOS’s  sampled 

b.  Frequency  of  evaluation 

c.  Sampling  procedure  and  size 

d.  Make-up  of  evaluation  team 

e.  What  data  is  collected 

5.  How  is  evaluation  data  summarized?  (mail,  on-site) 

6.  How  useful  is  the  data  from  the  mailed  questionnaires? 

7.  Who  are  the  results  sent  to?  (mail  and  on-site) 

8.  How  do  the  results  feed  into  other  ISD  Phases  (i.e.,  do  they  use  it)? 

If  not  used,  what  reasons  ore  given: 

a.  Timeliness 

b.  Data  available  from  other  sources 

c.  Format 

d.  Other 
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9.  What  role,  if  any,  do  SQTs  play  in  the  external  evaluation?  How  are  results  used? 

10.  Of  the  guidance,  procedures,  and  forms,  (get  copies)  used  by  your  people,  which 
would  be  especially  useful  to  others?  What  problems  that  you  had  in  past  were 
solved? 

11.  What  additional  guidance,  or  job  aids,  would  you  like  to  have  for  new  people 
entering  your  directorate?  Why?  What  problems  would  they  solve? 
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APPENDIX  D 


BACKGROUND  INFORMATION 
ISD  ACTIVITY  CHECKLIST 


Please  answer  each  question  as  it  pertains  to  your  current  work  and/or  previous  experience.  All  responses  will 
be  CONFIDENTIAL.  This  information  will  not  be  reported  on  an  individual  basis  to  assure  the  privacy  and 
anonymity  of  your  answers. 


1  Name _ _  2.  Grade 


3.  Date 


4.  Division _ 5.  Branch _ 

6.  Bldg.  # _ 7.  Telephone  H 

8.  Immediate  supervisor _ 

9.  Current  job _ _ _  _ 


10.  How  long  have  you  served  in  this  job? 


11.  Please  check  { \/ )  in  the  chart  below  aU  the  ISD  activities  in  which  you  have  had  some  experience. 


□  Preparing  critical  task  luting 

□  Conducting  field  validation  ol  critical  task  list 

□  Preparing  |ob  task  pcrloi mance  meaMrel 

□  Preparing  task  summary  sheets/iob  task  data  raids,  etc 

□  Preparing  list  ol  reference  materials  lu'  CM  SM 

O  Selecting  instructional  setting  (where  naming  is  to  be  conducted) 

□  Writing  training  objectives  ITLOs.  LOs.  CIOs,  and  LSsI 

□  Preparing  test  administration  manual  ITAMI 

□  Preparing  SQT  materials 

□  Prepar  mg  other  tests  (criterion,  etc.) 

0  Conducing  validation  ol  tests  by  subiec  matter  experts 

□  Conducting  held  validation  ol  tests 

O  Making  revision  (allowing  validation  tests 

□  Examining  student  entry  behavmr 

Q  Sequencing  ol  instructional  material  (course  mapl 
D  Clastilymg  learning  objectives  (elements)  by  typos  ol  'eatn.ng 


t  1  Making  media  methods  selections 
L)  Locating  selecting,  and  reviewing  existing  matenais 
IT  Developing  correspondence  sobcourse 
IT  Developing  TEC  matenais 
!  Developing  resident  lesson  plan 
1  Developing  P  F  sheets  (ivorkbOJkll 
.  J  Developing  sell  pared  performance  llessonl  guide 
IJ  Developing  Ah  scripts,  T  V  scripts,  etc 
f  I  Developing  programmed  text,  CAi,  etc 
L!  Developing  other  instruction  Iresibentjnonresident) 

IT  Conducting  developmental  testing  bone  on-one) 

IT  Conducting  imall  group  validation  (rials 
(  i  Making  revisions  following  small  group  validation  trials 
Li  Conductmg  '»tgr  group  validation  tu-ji* 

1  )  Other  I  speedy  I _ _ 


12.  Please  check  (>/)  in  the  chart  below,  all  the  ISD  activities  that  you  have  performed  o-.-  wili  perform  on  a 

sinQle  MQS.  ~~  ~  ~ 


D  Preparing  critical  task  listing 

O  Conducting  held  validation  ot  critical  task  lot 

Q  Preparing  |ob  task  pe'lotmance  measures 

O  Pttpermg  task  summary  sheets/|ob  task  data  cards,  etc 

Q  Preparing  list  ot  reference  materials  lot  CM'SM 

CD  Saltctmg  instructional  setting  (where  training  n  to  be  conducted) 

□  Writing  training  objectives  (TLOs.  I  Os.  CLOs.  ami  LSs) 

C  Praparmg  t*$t  administration  manual  ITAMI 

□  Praparmg  SOT  materials 

□  Preparing  other  tests  (criterion,  etc  I 

Q  Conducting  validation  ol  tests  by  subiect  matter  expem 
Q  Conducting  held  validation  ol  tests 
0  Making  revision  following  validation  lests 
O  Examining  student  entry  behevmt 
Q  Sequencing  ol  instructional  material  (course  map! 

□  Classifying  learning  objectives  (dementtl  by  tvsxrt  ol  learning 


!  1  Making  mad -a  methods  select'd -> 

l. !  Locating  ie'ect"<g.  and  rcvir -.wy  tv  s  ng  materials 

1  1  Developing  correspondence  v.Jhv> '*.< 

U  Developing  TEC  nsateriais 

l  1  Oevelop.ng  res  dent  lesvrn  plan 

(J  Developing  P  E  sheets  IvvorklHsoksi 

IT  Developing  sell  paced  per  tormaoce  (lesson)  guide 

l.)  Developing  AV  scums.  T  V  scripts,  etc 

t  5  Developing  jsrogrjmmed  text.  CAI.  etc 

(  1  Developing  othei  instruction  (revdent/non  resident) 

!  1  Conducting  developments'  testing  lone  on  ,>ne| 

l.)  Conducting  small  group  vahda.ion  trials 

(1  Making  revisions  following  imalt  group  validation  it  cals 

l  I  Conducting  iatge  group  validation  trials 

□  Other  (speclyl _ _ 
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ISD  ACTIVITY  CHECKLIST 


Please  answer  each  question  as  it  pertains  to  the  Instructional  Systems  Development  (ISD'  activity  you  have  worked 
on  during  the  past  two  weeks.  This  information  will  not  be  reported  on  an  individual  basis  to  assure  the  privacy 
and  anonymity  of  your  answers.  If  you  have  any  additional  comments,  please  enter  them  at  the  bottom  of  the  form. 

1.  Name_ _ _  2.  Date _ 

3.  Please  check  (/)  in  the  chart  below  the  ISD  activity  that  you  have  worked  on  during  the  past  2. weeks. 


□  Preparing  critical  task  listing 

□  Conducting  field  validation  of  critical  task  list 
O  Preparing  job  task  performance  measures 

□  Preparing  task  summary  sheets/job  task  data  cards,  etc. 

□  Preparing  list  of  reference  materials  for  CM/SM 

D  Selecting  instructional  setting  (where  training  is  to  be  conducted) 

□  Writing  training  objectives  (TLOs,  LOs,  CLOs,  and  LSsi 

□  Preparing  test  administration  manual  (TAM) 

□  Preparing  SQT  materials 

D  Preparing  other  tests  (criterion,  etc.) 

D  Conducting  validation  of  tests  by  subject  matter  experts 
D  Conducting  field  validation  of  tests 

□  Making  revision  following  validation  tests 
D  Examining  student  entry  behavior 

□  Sequencing  of  instructional  material  (course  map) 

G  Classifying  learning  objectives  (elements)  by  types  of  learning 


□  Making  media  methods  selections 

D  Locating  selecting,  and  reviewing  existing  materials 

□  Developing  correspondence  subcourse 
G  Developing  TEC  materia1* 

D  Developing  resident  lesson  plan 
G  Developing  P.E.  Sheets  (workbooks) 

□  Developing  self-paced  performance  (lesson)  guide 

□  Developing  AV  scripts.  T.V.  scripts,  etc. 

□  Developing  programmed  text,  CAI,  etc. 

□  Developing  other  instruction  (resident.'non  resident) 

□  Conducting  developmental  testing  (one-on-one! 

□  Conducting  small  group  validation  trials 

D  Making  revisions  following  small  group  validation  trials 
O  Conducting  large  group  validation  trials 

□  Other  (specify) 


4.  Did  you  complete  (at  least  in  draft  form),  ttre  activity  checked  above?  Yes. _  No. _ _ _ . 

5.  Estimate  the  total  number  of  hours  tfnat  you  spent  working  on  the  ISD  activity  checked  off  above.  _ Hours 

6.  What  percentage-  of  the  time  that  you  worked  on  This  ISO  activity  was  spent  on: 

(a)  Initial  Preparation  _  % 

(b)  rtevivWRevision  _ _  % 

(c)  Other  (explain)  _  % 


7,  What  problems  (if  any)  did  you  experience  in  performing  your  work  rt.  this  activity?  (Please  check  all  that  are 


applicable.) 

(a) 

Guidance  was  not  available 

(h) 

Input  to  my  ISD  work  was  unsatisfactory 

(b> 

Guidance  was  Inaccurate 

ii) 

Input  to  my  ISD  work  was  late 

(c) 

Gould  tot  get  naeded  infarmahm 

(j) 

Not  enough  time  to  gather  information 

<d> 

Information  was  Inaccurate 

_  (k) 

Not  enough  time  to  develop  material 

(e) 

References  were  tut  of  da** 

(f) 

Not  enough  time  to  review/revise  material 

Dkl  not  recvtvr  enough  supervision 

(m) 

1  did  not  receive  enough  tratmoy  to  do  this 

(g) 

Not  enough  help  from  other  per  onnei  _ 

work  satisfactorily 

(n)  Had  no  problems 


8.  If  the  input  to  your  ISD  work  was  not  satisfactory,  what  did  you  do? 


(a)  Reported  it  to  rr.y  supervisor 

(b)  Returned  it  for  revision 

<c)  Did  it  over  myself 

(d)  Not  applicable 

(e)  Other  (please  describe) _ 


9.  Was  the  output  (product)  from  your  work  returned  to  you  for  revision?  Yes. _  No. 

If  "Yes,"  what  had  to  be  changed?  _ 

(a)  Errors  had  to  be  corrected  _ 

(b)  Format  was  wrona _ 

(c)  References  were  wrc  lg  _ 

(d)  Incorrect  wording _ 

(e)  Documentation  was  inadequate  _ 

(f)  Other  (please  describe) _ _ 


COMMENTS: 


Pfette  ptac *  this  form  in  the  attached  envelope  and  place  «t  in  the  HumRRO  container  located  in 
Mr.  John  D.  OmUovich't  office  (BJcfg.  19704)  or  in  Mr.  John  R.  Myers'  office  (Bldg.  2S701). 

Thank  you  tor  your  cooperation. 


APPENDIX  E 


JOB  AID  FOLLOW-UP  INTERVIEW  FORM 


Interviewer 

Pate 


BACKGROUND  INFORMATION 

Name  _ _ _ _ _  Grade _ _ 

Division/CMF  ___ _ _ 

MOS  (working  on)  _ _ _ _ 

Current  Job  _ _ _ _ _  How  long  in  job? 

How  long  in  ISD  block  for  which  Job  Aid  designed?  _ _ 

Any  formal  training  in  this  ISD  block?  If  yes,  what  and  how  much?  _ 


Did  you  complete  a  Background  Information  Sheet  a  few  months  ago? _ If  no,  have 

them  complete  a  Base-line  Data  Background  Information  Sheet. 

Who  is  your  supervisor?  _ _ _ _ 


FLOWCHART 

1.  To  what  extent  did  you  actually  use  the  Flowcharts? 


2.  Were  the  symbols  used  understandable? 


3.  Was  written  material  on  side  understandable  and  useful? 


45 


FLOWCHART  (Continued) 

4.  Were  there  times  when  you  were  referred  to  the  manual  unnecessarily?  Times  when  you 
should  have  been  referred  and  were  not?  Which  blocks? 


5.  Did  you  ever  go  to  the  Manual  to  see  an  example  when  not  required  to  do  so?  If  yes, 
how  frequently? 


6.  Was  there  a  need  for  expansion/collap&e  of  some  flowchart  blocks?  If  yes,  where? 


7.  Were  there  too  few/too  many  supervisor  checks  of  your  work  called  for  in  the  aid? 
Where  should  they  have  been  added  or  deleted? 


8.  What  is  your  overall  evaluation  of  the  flowcharts?  How  can  they  be  improved? 


INTRODMCTTON  TO  JCBAIpS 


9. 


How  good  a  job  did  It  do  in  explaining  what  the  dob  Aids  were?  How  can  it  be 
Improved? 

m 


RESOURCE  MANUAL 


10.  Did  you  use  the  Resource  Manual?  How  well  did  it  cover:  CODAP,  School  Conducted 
Field  Surveys,  and  School  convened  panels? 


MANUAL 


11.  Were  the  partial  flowcharts  at  the  top  of  the  page  useful?  To  what  extent  did  you 
use  them? 


12.  Was  the  overview  to  the  job  aid  useful  to  you?  How  can  it  be  improved? 


13.  Were  the  questions  used  to  introduce  each  area  useful? 


14.  Is  the  text  in  the  manual  too  difficult  to  read?  Explain. 


15.  Were  there  inaccuracies  in  the  information  presented?  Explain. 
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MANUAL  (Continued) 


16.  Were  important  sources  of  information  omitted?  If  yes,  what  are  these  sources 


17.  Was  inappropriate  or  obsolete  information  presented?  Explain. 


18.  How  useful  were  the  references  provided?  Were  they  correct?  What  additional 
references  or  sources  of  information  do  you  usually  use? 


19.  Kow  useful  are  the  examples  provided? 


WORKSHEETS  (Continued) 

22.  Did  the  Worksheets  require  too  little/too  much  documentation? 


GENERAL 


23.  Was  the  reading  level  of  the  Job  Aid  (flowcharts,  introduction,  etc.)  too  high/ 
toe  low? 


24.  What  did  you  do  before  you  had  this  Job  Aid  that  was  better  or  more  efficient 
for  performing  the  work  in  this  area? 


25.  Do  you  think  that  by  using  this  Job  Aid  you  have  turned  out  a  better  quality 
product  than  without  the  aid? 


26.  In  the  long  run,  do  you  think  that  use  of  the  Job  Aid  will  turn  out  products 
faster  or  slower  than  without  the  aid? 


27.  Do  you  think  the  Job  Aid  Is  too  long/too  short  or  about  right  in  length? 


GENERAL  (Continued) 

28.  Will  you  continue  to  use  this  aid  in  the  future?  If  no,  find  out  why. 


29.  What  type  of  person  do  you  think  would  find  this  aid  useful?  (i.e.,  inexperienced/ 
experienced) 


30.  In  order  to  use  the  Job  Aid  you  needed  materials  from  individuals  who  had 
performed  ISD  activities  preceding  yours.  Did  the  materials  used  hinder 
or  help  you  in  your  work? 
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